Evidence based digital training portfolio

ABSTRACT

A system for collecting and evaluating data records comprises a configuration device associated a data collection device associated, and a data monitoring device. The devices are registered with a server. The data collection device transmits a task to the server over a network. The data collection device receives the task from the server and records a data record in response to the task. The data collecting device groups the data record with metadata to produce a verified data record and transmits the verified data record to the server. The data monitoring device receives the verified data record from the server and facilitates the evaluation of the verified data record. The evaluation of the verified data record is used to generate an updated verified data record comprising evaluation results and updated metadata and is transmitted to the server and added to a portfolio.

FIELD OF THE INVENTION

The present disclosure relates to evaluating the performance ofstudents, employees, and professionals using a verified, evidence basedapproach.

BACKGROUND OF THE INVENTION

Currently the licensing of professionals such as doctors, surgeons,engineers and others, occurs across several years of course curriculum,technique practice and perhaps simulations. In particular, hand-on,practical exams are difficult to assess. These sessions are observed byan examiner or teacher and rated qualitatively by that person. Afterseveral years of training and education a paper based record with thecumulative comments of all teachers is submitted to a licensing orcertification authority. The decision to grant a license is measuredagainst a set of standards based on the qualitative description of anapplicant's training as well as on the quantitative grading system used.

There are several drawbacks of this traditional approach. One drawbackis that the authority granting the license or certification may neverhave actually observed this professional's techniques nor expertise andis basing the decision mostly on the qualitative assessments of thestudent's teachers and examiners.

Another drawback is that for a class of many students there are likelyseveral different examiners. Each examiner will have slightly differentstandards and experience and will grade students differently using theirown set of different subjective standards.

To ensure consistency and fairness, practical examinations may berecorded or videotaped by various methods. However, this approach hasthe limitation that it is difficult to ensure the content veracity ofthe recording. Digital and analog files may be altered and edited andthis tampering can be very difficult to detect. Tests, questionnaires,and surveys may be accessed surreptitiously in advance. As well,evaluation results may also be tampered with. To ensure a fairevaluation, a licensing or certification authority must be sure thatrecordings and results have not been tampered with and that it shows theperformance of the actual student performing the procedure or exam onthe date indicated.

BRIEF SUMMARY

An exemplary embodiment of the invention includes a system forcollecting and evaluating data records. The system comprises aconfiguration device associated with a first user. The configurationdevice is registered with a server and transmits a task to the serverover a network. A data collection device is associated with a seconduser. The data collection device is registered with the server andreceives the task from the server and records a data record in responseto the task. The data collecting device groups the data record withmetadata to produce a verified data record and transmits the verifieddata record to the server. A data monitoring device is associated with athird user. The data monitoring device is registered with the server andreceives the verified data record from the server and facilitates theevaluation of the verified data record. The evaluation of the verifieddata record is used to generate an updated verified data recordcomprising evaluation results and updated metadata. The data monitoringdevice transmits the updated verified data record to the server. Theserver stores the updated verified data record in a portfolio.

In other embodiments of the invention the system, the identity of theconfiguration device and the first user is verified before allowing thetransmission of the task to the server. The identity of the datacollection device and the second user is verified before allowing thereception of the task from the server and before allowing thetransmission of the verified data record to the server. The identity ofthe data monitoring device and the third user is verified beforeallowing the reception of the verified data record from the server andbefore allowing the transmission of the updated verified data record tothe server.

In some embodiments of the invention the metadata comprises a pluralityof parameters, the parameters being used to confirm that the verifieddata record is an accurate representation of the data record.

Data may be divided into a plurality of segments to be transferredbetween the server and, the configuration device, the data collectiondevice, or the data monitoring device, the receiving device confirms theveracity of each of the plurality of segments as it is received. A firstalgorithm is used to determine the veracity of each of the plurality ofsegments and a second algorithm is use to determine the veracity of allof the plurality of segments after all of the plurality of segments havebeen received.

The metadata may comprise a plurality of parameters, that may be used toconfirm that the verified data record is an accurate representation ofthe data record.

The data record may comprise video recordings that may be used by thethird user to evaluate the emotional or the physical state of the seconduser.

The server may comprise a plurality of content storage locations thatimplement a distributed file structure wherein the portfolio is storedin more than one of the plurality of content storage locations.

A second exemplary embodiment of the invention 9 includes a method forcollecting and evaluating academic performance. The method comprisesregistering a student and a personal device with a server where thestudent is associated with the personal device. Also, registering aninstructor and an instructor device with the server where the instructoris associated with the instructor device. As well, registering aninstructor and a monitoring device with the server where the instructoris associated with the monitoring device. The instructor device combinesa task with task metadata to produce a verified task and transmits theverified task to the server. The personal device receives the verifiedtask from the server. The personal device records the performance of theverified task to produce a data record. The personal device combines thedata record with data metadata to produce a verified data record andtransmits the verified data record to the server. The monitoring devicereceives the verified data record from the server. The monitoring deviceis used to record an evaluated data record and combines the verifieddata record and the evaluated date record with evaluation metadata toproduce a verified evaluated data record and transmits the verifiedevaluated data record to the server. The verified evaluated data recordis combined with student results and added to a portfolio associatedwith the student.

Further embodiments comprise registering an authority and an authoritydevice with the server. The authority is associated with the authoritydevice and the authority device receives the portfolio from the server.The authority device is used to record an overall evaluation of thestudent's performance. The overall performance is combined withperformance metadata and with the portfolio to produce a verifiedportfolio and is transmitted to the server.

Other embodiments comprise the authority device being used to record anoverall evaluation of the instructor's performance. The overallperformance is combined with performance metadata to produce a verifiedinstructor evaluation and is transmitted to the server.

The foregoing and additional aspects and embodiments of the presentdisclosure will be apparent to those of ordinary skill in the art inview of the detailed description of various embodiments and/or aspects,which is made with reference to the drawings, a brief description ofwhich is provided next.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other advantages of the disclosure will becomeapparent upon reading the following detailed description and uponreference to the drawings.

FIG. 1 is a depiction of the workflow process for the analysis of theeducation and training performance of student enrolled in the program.

FIG. 2 is a depiction of the purpose-built technology components thatmake the execution of the workflow possible.

While the present disclosure is susceptible to various modifications andalternative forms, specific embodiments or implementations have beenshown by way of example in the drawings and will be described in detailherein. It should be understood, however, that the disclosure is notintended to be limited to the particular forms disclosed. Rather, thedisclosure is to cover all modifications, equivalents, and alternativesfalling within the spirit and scope of an invention as defined by theappended claims.

DETAILED DESCRIPTION

Embodiments of the invention are designed to capture the actualdemonstration of competency in the execution of a procedures orprotocols as described by relevant licensing of and certificationauthorities. This system captures the evidence or proof required tomeasure and effectively evaluate a student's success in achieving thoseperformance criteria through data and video analytics of contentcaptured live. The content recorded or captured is used to replacetraditional qualitative assessments with replay and searchable contentcapturing the actual demonstration of procedures or protocols. Thesystem provides a verified and trusted historical record of the trainingresults of an individual or group of students to be collected, saved andreviewed. The system provides actual evidence of student's proficiencyin certain techniques and irrefutable evidence of the student'sprogress. The audio/video content is enrichened with digital data,notes, observations, and other measurements specifically designed tomeasure and assess the students' performance.

The system provides capabilities to edit and redact the captured contentas necessary to best present the key factors within a procedure orprotocol. The results are stored in a secure training portfolio and mayinclude digital recorded audio, video, references, text, annotations,and any other type of digital media. The training portfolio alsoincludes permissions and access restrictions, an audit trail and digitalsignatures to ensure the veracity of the information. This could also beachieved in a number of different ways such as through a file structureor database only version. The key concept to provide all of the evidenceof achieving competency in a particular skill is that all of therecorded data is available as a portfolio or unit for submission to thelicensing authority.

The portfolio in stored in a secure, confidential manner and may bepresented to the licensing or certification authority to provideevidence, with veracity, to the practical skills and performance of anapplicant. The authority is able to observe and evaluate the actualtechniques developed as if first-hand and thus provide an accurate andconsistent mechanism for assessing an applicant's knowledge andtechnique versus the prior methods. Results may also be observed andregraded at a later date to ensure correctness and fairness. It may alsobe used to identify and eliminate variations in examiner assessmentsintroduced using a traditional qualitative assessment. A completegrading and test history is also included as well as data analytics forcomparisons to other classes, faculties or educational institutions. Inembodiments of the invention all of the content is maintained as asingle portfolio, it could however be captured as separate files whichare then assembled when ready to submit. These files could also bestored on other digital media such as Blu-ray, DVD or other types ofdigital portable media.

In order to illustrate the exemplary embodiment, FIG. 1 shows an exampleof the workflow process for the analysis of the education and trainingperformance of student enrolled in the program. The student or studentsmay be students studying a profession studying a discipline thatrequires hands on practical exams to demonstrate competence. In thisembodiment they may be medical students studying a program that may spanseveral years. however, embodiments of this invention may equally applyto any type of student, employee, teacher, professor, driver, pilot,police office, or any type of study or activity that includes apractical component that may have audio, video, or other digital contentto be evaluated. It may be used to evaluate the student as well as theauthority for fairness and consistency.

In this embodiment there are three roles in the exemplary embodiment;the student, the examiner and the supervisor. Though they will all bereferred to in the singular, each or all of the roles, or parties mayalso be a group of people. The student may be a medical student startingtheir studies in a multi-year program. The examiner may be one or manyinstructors and professors who evaluates the written and practical workof the student over the course of their studies. The supervisor is theperson or person who will review the evaluation produced by the examinerto determine if the student is successful at their studies and isqualified to practice at the end of the multi-year program. In someembodiments, the examiner or the supervisor role may not be required. Inother embodiments the supervisor may also be a licensing oraccreditation board.

The student, examiner, and supervisor all interact with the systemthrough specialized software or an app installed on their mobile device.A student will typically access their results through a cellularsmartphone that the majority will already own and be familiar with theoperation of. Alternatively, the student may choose to access the systemusing any of a variety of digital devices known in the art includingtablets, laptop computers or desktop computers. Students may alsodownload exams, tests, and questionnaires, complete them, and upload theresults.

Examiners and supervisors will typically access the system using atablet or other digital device. Examiners and supervisors may defineproprietary assignments, tests, exams, and questionnaires, setpermissions, access rights, and security levels and upload them to thesecure server. This includes indicating which students, sections, orclasses may assess the material as well as times when the exam may bedone, and how long the student has to complete the work. They may alsospecify the contents of the results such as the requirement to include avideo of the procedure, an answered questionnaire, and approval from andexaminer.

Referring to FIG. 1, In this depiction, the process begins with theexaminer 102 working with the Supervisor 101 to prepare demonstrationvideos of the intended procedure or protocol to be taught. TheseAudio/Video demonstrations are captured in an appropriately outfittedroom and once captured are edited to exemplify the best technique forperforming the procedure or protocol 103. The demonstration audio andvideos are submitted with appropriate security credentials and classdefinitions to the secure web collaboration portal 104 along withreference materials 105 and proprietary tests 106. Specific content isclassified with access permissions in such a manner as to only permitstudents registered in a particular term and class to be able to accessthe content.

Content is saved in a database 107 including data structures and storageprotected by digital security techniques to prevent corruption or accessby unknown entities. The database can use any one of many techniques asare known in the art including SQL databases. All access andmanipulation of the content is tracked and recorded. The audit trail ofactions and events can be reviewed by an administrator securely asrequired.

Students are provided with a mobile device interface 109, typically asmartphone, that permits secure registration and subsequent access tocontent utilizing credentials and biometric authentication. For eachstudent, a digital portfolio is setup which is maintained throughout thecourse of their training and education. This secure digital portfoliocontains all data related to the student's program. Students, examinersand supervisors all use appropriately equipped rooms 108 for audio/videocapture of sample procedures or execution of protocols. These roomscontain the equipment and tools needed for each to achieve the captureof demonstration procedures or protocol execution as appropriate. It isenvisioned that this part of the process could also be achieved by usinghand-held video devices to capture practice sessions in any locationwithout necessarily having a purpose built room or facility. Althoughthis would be less effective, it would achieve a similar end.

The registered students using authenticated devices who log on usingbiometric identification are now able to access the content available onthe secure collaboration portal specific to their current term. Thestudent prepares the demonstration procedure audio and video content inthe room 108 and edit the results as appropriate and as allowed by thesystem. The student may also complete an associated test orquestionnaire related to the specific procedure or protocol beingpracticed. The procedure may be repeated multiple times. Once satisfiedwith results the student submits these materials to the securecollaboration portal 107. The system creates, updates, and maintains ametadata file which is transferred along with the content to preservethe veracity of the submission. The transfer themselves are alsoprotected by encryption techniques as are known in the art.

The examiner 102 is alerted that the student has submitted new contentand can review it at any time. The materials submitted by the studentare reviewed and a grade is assessed. These grades along with comments,additional reference materials or more sample audio and video content isreturned to the secure collaboration portal within the student'sportfolio in the database 107. As the volume of content within eachstudent's portfolio reaches a sufficient amount data analytics can beapplied. These data analytics use statistical methods to measure theperformance of the student against established standards in real-time. Adashboard is presented to the student which can display statisticsalerting the student to their performance on a specific procedure orprotocol compared with standards, other students, other classes andhistorical parameters. The student will also know immediately how theirperformance compares to others. Likewise, the examiner has a dashboardthat depicts their class performance against similar statisticalmeasures. These data analytics will also identity students who arefalling behind and provide for remedial actions. The Supervisor 101 hasa similar dashboard which provides a broader scope across all classesand perhaps the school.

The data analytics provides a statistical mechanism for measuringreal-time performance of the student in an educational and trainingscenario. The benefit of knowing how one is progressing affords bothstudent and examiner to access remedial actions specific to their areaof difficulty as soon as it is identified. These educational andtraining performance measures will substantially improve both theefficiency and effectiveness of training techniques and allow for a muchfiner degree of detail in assessments backed by audio and video of theirperformance. This area of data analytics may be used in guiding theperformance of the student as they grow more competent, improving theirskills to a level acceptable to the institution. The data produced bythe data analytics could be generated using numerous algorithms foradvanced analyses and much finer guidance of skills competency. Thiscould be achieved with devices designed specifically to capture musclecontrol, heart-rate, body temperature and perhaps neural activity. Thesedata results could be used to quickly identify if the student isadvancing in certain new ways measured when compared with the progressof the best students. Many performance criteria could be used to achievesimilar or more detailed training results.

Material and results are added to the student training portfolio untilthe program has been completed. At this point the student's digitalportfolio is assessed by the examiner and supervisor 110 forrecommendations on applying for license or certification from thegoverning body 111. Should they decide to proceed with the applicationthe governing authority is provided with the complete multi-term digitalportfolio for each applicant. The governing body now has the opportunityto assess through actual viewing of the student's techniques via thesubmitted audio and video and other content provide in the student'sportfolio. This digital portfolio allows the governing authority todirectly assess the skills and performance of the student directly firsthand. In this manner the variations introduced by qualitativeassessments by third parties is eliminated and a much more robuststandard of performance measures applied. Once the governing body hascompleted its assessment the results 112 are provided and availabledirectly to the student on their registered device. Furthermore, theresults may be reviewed later in the case of appeals, remarking, orother review or auditing processes.

The technology depicted in FIG. 1 also ensures that the content producedis captured, transferred, stored, and viewed in a secure manner and isfully audited such that no editing, changing, addition, corruption oraccess by unknown parties can be perpetrated. Embodiments of theinvention provide the veracity required to depend on the student'sdigital portfolio when reviewing performance, assessing technique,assigning grades or applying for a certification or license as a resultof demonstrated performance. This is achieved in part by thetechnologies, mechanisms and algorithms used to protect the veracity ofthe content, authenticate the devices, people and authorize theregistered participants. To achieve this, every device is registeredinto the system with a unique system identification code referred to asa node ID. Every participant is also registered into the system with aunique and vetted set of credentials. Every piece of content created istracked through a collection of parameters or metadata which can be usedto demonstrate that the content being viewed is true to the originalcaptured content and has not been corrupted or changed in any way. Thatauditing tracks every time that piece of content was touched by anyprocess, account, device or program. All content is encrypted duringtransit and certified to be an exact copy of the original between sourceand destination. Where appropriate, metadata contains digital signaturesand other identifying factors to link credentials to content produced.Maintaining a level of security and data protection is important toauthorities relying on the student training portfolio to issue licensesor certifications in a particular skill or vocation. Achieving a levelof security sufficient to withstand all potential threats could beachieved in a number of advanced ways. Verification could be included asactual data overlaid on all content such as might be applied on videorecordings. The use of time stamps, GPS locations and biometric datacould be applied on the actual content and consequently also providedata verification. Other mechanisms could include certification ofcontent using holographic images such as are used on some currencies.

Other sources of data are also used to add digital media to theportfolio. These include audio/video recordings, with the possibility ofmultiple views being recorded simultaneously. The recording may also besupplemented with real or simulated recordings from medical devices thatrecord respiration, pulse, blood oxygen level, and a number of otherdevices.

The digital portfolio is stored on a secure server which may be aconventional server or a cloud storage services and may be accessedthrough any number of wired or wireless communications networks as areknown in the art. The secure server may also store course material,tutorial material, schedules, exam papers, assignments, questionnaires.Proprietary or confidential material will be encrypted. It may alsostore links to other material and databases that store supportinginformation such as university registrations and course schedules.

The system may be configured with access permissions by a systemadministrator. The student will be permitted to access their ownresults. An examiner may access the results of any student for the testsor procedures that they observed or evaluated. Supervisors are able toview the results of any of the students where they are required toevaluate their portfolio to determine their overall success or failurein a program. Dashboards and statistical summaries of results are alsoaccessible to users and will often be customized for the needs andpermissions of those individuals. Permissions may be configured andupdated by the educational institutions and licensing bodies to meettheir needs.

The system utilizes a number of techniques to establish the permissionsassociated with each party. These are developed to ensure that a studentin course during a term program is properly registered and as a resulthas access to materials and test associated with this particular segmentof the course and program given their current stage. Students will haveaccess to all of the materials for a particular segment of theirtraining and education when appropriate and in a timely manner. As aresult, the permissions must change as the student progresses throughthe program. In addition, the permissions are designed in such a way asto allow different versions of some materials to be issued to studentsin the same course section at the same time. This would be to providethe examiner the ability to have slightly different tests orquestionnaires sent to groups of students at the same point within aprogram. Permissions are also established to provide for students tobelong to a group who have the same set of permissions. Thesepermissions could modify search and retrieve parameters to restrictstudents in one class from retrieve materials provided to another class.The system delivers algorithms that may also restrict the depth of asearch parameter to limit the breath of a search from a systemperspective.

When the student starts their studies, their preferred digital devicewill be registered with the system and a digital portfolio is created.Their digital portfolio may initially be populated with digitizedversions of their application results. Application results may include,but are not limited to, transcripts from previous program, entrance examresults, letter of reference, and interview results. In the case ofnon-written material such as an audio recording of a conversation or avideo recording for of an interview, a digital audio or video recordingwill be included in the portfolio.

As the student progresses in their studies, results of course work,assignments, questionnaires, written and practical exams will bedigitized and added to the portfolio. At the end of their studies, theportfolio becomes digital evidence of the courses, programs and testscompeted, and the skill in all aspects of those studies.

When a practical examination is performed, such as an orthopedic surgeonstudent demonstrating their surgical technique, the examination may takeplace in an operating room equipped with video cameras, microphones, andother recording devices. At the start of the exam the student may accessthe secure server and access instructions, demonstration videos,questionnaires and any other material as specified by the examiner orthe supervisor. As the student performs the procedure, it is recorded.Multiple views may be recorded simultaneously such as a view showing thestudents hands as they perform the procedure and another view showingthe entire scene. The recording may also be supplemented with real orsimulated recordings from medical devices that record respiration,pulse, blood oxygen level, and a number of other readings. Whereallowable, an exam may allow a student to perform a procedure multipletimes. The student will be able to review the results of each repetitionof the procedure and choose the one to submit for evaluation.

The examiner may observe the procedure live or may view the digitalresults in another location or at a later time. The examiner willevaluate the student's performance and record a grade together withcomments. These comments or notes can be applied while the recording isprogressing and could be added to the content as public or privatenotes. In these cases either private for the examiners use or publicsuch that the student can also see the comments. This type of live noteduring the demonstration of the procedure would allow for importantpoints to be annotated when they occur during the practice procedure.These notes could be reviewed by either or both Examiner and Student asa way to improve performance the next time in those specific areas. Theycould also be used as high-quality notes post-practice for referenceprior to another attempt. The grade and comments will be combined withthe recordings and other metadata such as the student's identity, thetime and place or the test and this will be saved in the portfolio. Foreducational institutions, metadata can also include the course, sectionand other data and results may be organized and grouped by each course.

Information in the portfolio is often private and proprietary to thestudent or the institution or licensing body. One example arequestionnaires that often accompany practical exams and are oftenproprietary to the educational institute or hospital that administersthe exams. The questionnaire may be downloaded from the secure server tothe student's device. Security features of the student's softwareprevent the copying of the questionnaire and the saving of thequestionnaire on the device. Security features that prohibit copying andlocal storage may include storing the questionnaire only in memory inorder to prevent the questionnaire being stored in non-volatile memoryon mobile device. Similar results could be achieved by directing thestudent to log on to another secure web portal specially to view theappropriate test or questionnaire. In this model the proprietary test orquestionnaire never leaves the institution's digital domain.Alternatively, this could also be through the inclusion of digitalrights associated with the proprietary documents.

FIG. 2 depicts six primary technologies which work togethertransparently to execute the workflow and result in a high-quality,advanced and secure digital portfolio of a student's education andtraining performance.

Computing devices 201, 202 are advanced capture nodes running“satellite” software on tablets, suitably configured laptops,workstation, or other computing devices. These computers run a suite ofsoftware solutions designed to capture digital content that includesaudio, video, data, text, images and other multimedia. These managementstations provide functionality for editing the content, adding contentto the captured data, and redacting the content should that benecessary. The satellite workstation provides hundreds of functions formanaging precisely how this content is configured and managedlonger-term. During the capture of the relevant content the digital datais securely moved in real-time to other content servers 205 forlonger-term storage. The local satellite workstation retains the contentfor a period of time determined by the operator or administrator througha retention schedule feature designed for that purpose. Protectivefunctionality is in place to ensure that content is not deleted until acertified copied is present on a content server. The Examiner 201 andtheir Supervisor 202 are equipped with compatible satellite workstationsto provide the ability to remotely monitor capture events on anothernode or search for content for performance assessments.

Students registered in this educational and training program areequipped with mobile applications 203 with corresponding interfaces thatprovide an easy to use, simple, and intuitive program for moving contentand collecting results and performance assessments. These devices arealso used to access a digital dashboard that uses data analytics toprovide statistics regarding their progress in the program from a numberof different perspectives. The mobile interface is the primary tool bywhich the student interacts with the educational training program. Thisprogram and device are used to authenticate and authorize the registeredstudent to access and move materials and content as required. Studentsare authenticated through a number of algorithms including biometricidentification, student and node registration parameters, as well assecure credentials. All content is maintained in a secure and separatearea of the smartphone 203 and is encrypted in transit. A programmaticalgorithm is used to ensure that the correct content is moved to thecorrect student at the correct time. All content is tracked and sentalong with content as it is moved between the student device 203 and thesecure collaboration web portal 205. Students, Examiners and Supervisorsalike will also use a satellite workstation situated in proximity to thepractice room 204. This room is suitably equipped with multiple cameras,microphones and other devices to record and capture their proceduraldemonstrations. This equipment is used to capture the actual procedureas it is being executed as an example of how it is best done by experts.The practice room would also be used by students to practice and recordtheir effort at the procedure for later submission for performanceassessment. Once this content is captured it can be edited and modifiedas appropriate for submission from the student's mobile device 203. Allcontent which is captured by the system is tracked and data added atevery stage regarding its veracity and history. Content is encryptedduring transit and is secured when at rest. Options are provided to havedata encrypted at rest as well.

Content is moved between the examiner 201, supervisor 202 and student203 via a secure collaboration web portal and server 205 that ensuresthat only those parties authorized and authenticated can access thecontent and that that content is specific to the term that the studentis currently enrolled in. This application is designed to provide asecure collaboration experience while providing as much ease of use atany time and from anywhere. This portal is also used to providematerials for the course to the student which would include videos,audio, references and specific tests associated with the student'scurrent stage on the training and education program. These materialscould include schedules, practice room assignments, social media items,references, reference materials, notes, instructions, curriculum as wellas items of interest relevant to the program registrants.

In order for the system to work the veracity of all digital items storedin the portfolio must be trusted. Users of the system must be sure thatnothing has been altered, deleted, added, and that the data is correct.To ensure the veracity of the data stored in the portfolio the systemsupports a number of features. These include registering and restrictingthe devices that can access the system, recognizing devices or nodes andspecifying the roles of individuals. All digital data is accompanied bysecure metadata requirements for ensuring the veracity include ahistorical audit trail that tracks all creation, additions, edits,deletions, and any other relevant operations on the digital data. Thesetechnologies have been developed as a secure and integrated suite oftools that ensure that content, people and credentials are protected atall times. They produce a clear audit trail and clear authenticationthat the materials submitted are from an authorized participant and thatthe secure digital portfolio can be referenced as a true representationof the students educational and training performance. Alternately, theuse of advanced encryption techniques utilizing two-factorauthentication along with a biometrics technique such as irisidentification could also be used to maintain a secure package withoutnecessitating the collection of data. With this technique the veracityof the documentation is entirely dependent on the strength of theencryption technologies but also highly dependent on the definition ofsecure process rules to guide the human interactions.

Designs of these components support the exchange of system metadata tovalidate the capture, editing, redaction and movement of data betweennodes. These exchanges occur as each component establishes a connectionto the next to exchange information. The results of that exchange arecaptured should later examination of the veracity of content berequired. This trail of audit data is maintained throughout the life ofthe digital portfolio such that is legitimacy can be established at anytime. Other mechanisms of exchanging data could be used such as secureFTP transfer or secure replication design. These would be less resilientto network failure but would achieve a similar end result.

The system utilizes a synchronization technology that provides a securemechanism for copying data from one location to another with assurancesthat the copy is identical to the original. This happens in real-timeand includes a number of techniques to ensure that the authenticity ofthe copy can be established and that the transactions involved in thattransfer are recorded. The system begins to transfer packets of dataimmediately after the beginning of the capture process, which could bewithin as soon as several seconds. Connections, established for thepurpose of synchronization, are authenticated through the use of thesystem provided node registration ID and other software licenseparameters. These parameters are verified every time a new connection isestablished. Once this has been established an encrypted pipe is set upto allow synchronization within a TLS (Transport Layer Security) levelpipe. When the encrypted pipe has been established, packets begin toflow to the destination, in most cases a networked server or, cloudserver, or cloud storage. A digital signature is established at theorigin and at the destination and at frequent periods, these arecompared to confirm that the data sent since the last check isidentical. This mechanism establishes that the content at thedestination so far has not changed from the content at the origin andthat synchronization continues properly.

Device registration is a key component for ensuring the security of thesystem against cyber-threats and certifying that the source of anycontent can be positively identified. When the system is firstimplemented and also with any subsequent device additions thetransparent management process initiated by central management serversoftware generates a unique node identification signature that isapplied when any new device is added. This signature is generated oncecertain other criteria are confirmed, such as license state, networkaddress, and other relevant parameters. Every generated signature mustbe unique within a system regardless of how large an installation,geographic location or network segment. Each customer installation hasunique identifiers to protect against the same identifier occurringacross independent systems. Identifiers are unique to the specificdevice and expire if and when a device is replaced or retired. Theseidentifiers or codes are confirmed every time a process connects to aspecific device for interaction with the system. No secure process willcontinue if the code confirmation fails. Another design to achieve asimilar outcome would require the use of more biometric data toauthenticate the user regardless of device used. This would also workbut would provide less overall identification data in terms of location,device used or other detail connection parameters.

A separate device or node registration process occurs at initialimplementation or anytime a new device is added to the existing system.This registration process provides the “digital signature” necessary forthe system to authenticate the device every time a request is made for aprocess connection. The authentication of the device at each processconnection verifies a number of factors including but not limited to,are licenses active and do they permit this connection, are both ends ofthe connection authorized to make this connection, are both end devicesauthenticated, what credentials are being used to create this connectionand so on.

Audit trails were developed to ensure that should the content beingcaptured be used as proof or evidence, that certain conditions,performance criteria or events took place. By maintaining a history ofthese for every piece of content the veracity of that evidence would becertified and validated by being able to produce the historical eventsoccurring with a piece of content from the time it is first captured tothe time it is used to support the application for a license orcertification as a result of professional education or training.Capturing evidence that a professional has performed to a certain set ofstandards after training and education could be tampered with. Audittrails provide direct evidence that this is or is not the case. Thesetrails are system generated and do not allow for user interaction. Ahistory is maintained with every profile and in some cases is protectedwith encryption as an additional layer of security. Histories aremaintained in multiple places and thus are hard to re-constitute withouta detailed knowledge of how that structure is designed. These audittrails are chain-of-custody files in the legal sense as the potentialthat content could be used in a legal action would require suchveracity.

Audio files are recorded in a proprietary format to further protectcontent from simply using a commercial available player to play contentwithout authorization. It is possible to export this data to a playableformat but that is done where authorized users want to review contentoutside of the system. A record of that event is captured by the systemand becomes a part of the audit trail discussed above. The structure ofa record is proprietary to the system and must be played using thesystems own playback capabilities to occur in a recognizable manner.These proprietary formats can be further protected with at restencryption technologies. Portfolio records are maintained in numerouslocations and structures and must be re-assembled to provide a replay ofthe content. This is particularly important if replaying only portionsof the captured content. Moving to a specific time within the content isdone utilizing a novel algorithm coded within the solutions players. Theformat must cover the multiple different digital components that make upa digital portfolio. It is critical that the audio format maintain achronologically accurate integration with video, notes, digital signalsand other digital data. That integration covers multiple channels ofeach. The audio format must maintain accurate time encoding such that onreplay the portfolio can re-create the same conditions as were observedwhen the LIVE recording occurred. The format of the audio structure isconducive to this requirement. Because audio tracks are recorded asseparate channels and can be recorded as a mix with video the audiofiles must contain metadata which defines how it relates to otherdigital data. In this instance the AAC audio produces high qualitymulti-channel sound and can be combined in various container typesincluding both Apple and Microsoft MP4 containers.

Details on the mobile application are important as the content capturedmust provide other metadata to prove performance of a procedure orprotocol against certain established standards must be protected againstunauthorized tampering. At every step from the point of capture to thepoint where a review is being conducted to access eligibility for alicense or certification the system must rely on digital methods andtechnologies to certify that this evidence of performance being providedin a student's portfolio is true and unchanged from the live captureevent. To do that the system must protect and document the capture ofcontent in many ways. Primarily these include the camera, microphone,video, instrument or device being used to capture content to the localrecording device. That initial transit must be protected against outsideinterference and as such an encryption technology is used. Once thecontent is at the local capture device that content must be protectedfirst while it is resident on the recording device and secondly as it ismoved to the local content server called a mid-tier server. Whileresident on the local recording device it is protected via digitalsignatures and other credentials. At this time the content is stillincomplete and in transition. Once complete is can be protected withat-rest encryption. This content is contained across multiple files andmultiple structures and is maintained with rich metadata. Digitalsignatures are used to certify that the stored content is unchanged fromthat recorded originally. This solution provides multiple mechanisms forprotecting the captured content at every stage during the building of aneducational and training portfolio. These mechanisms include the use oftools such as TLS level encryption, SSL, one and two way certificatesunder a PKI structure, the use of CRC checks and digital signatures andstorage mechanism which provides a distributed structure.

Data protection mechanisms provide a level of cyber-security that allowsfor content to be securely and safely stored using either private orcommercial Cloud technologies for storage. The content can be encryptedusing AES-256 or TLS level encryption along with the use of secure pipesusing SSL or HTTPS connections provides secure movement of data to andfrom cloud storage. Furthermore, the audit trail captures all events andactions that occur with the data and as a result the veracity of thecontent can be maintained regardless of where it is stored. For certainsituations the use of secure cloud storage is chosen, particularly wherethousands of educational and training portfolios are concerned. The useof the secure collaboration web portal requires that digital content bemoved back and forth between examiner, student and supervisor. As aresult, that content must be protected regardless of the mechanism usedfor storage. The use of secure cloud storage also makes sense whenlooking at long term archive storage of educational and trainingportfolio's. A particular mechanism that is used to protect the digitalcontent against cyber-threats is the use of a distributed architecture.This provides for creating multiple identical authenticated contentstorage locations in real-time. This technique is further reinforced bythe use of the distributed file structures used to store the content. Noone file represents the entire educational and training portfolio tore-constitute the portfolio the systems internal algorithms must beused. At any one time multiple authenticated copies of the digitalportfolio will exist across the multi-tiered architecture. Not only doesthis technique prevent possible corruption but also allows forre-building the content should one location be compromised. Thesetechniques are critical and provide a degree of insurance that thecontent has veracity and will deliver evidence of the studentseducational and training performance in certain procedures and protocolsshould a legal challenge be launched. In some embodiments, the studentportfolio could be contained on secure portable media such as a USB keydesigned specifically for the purpose. In this model the key or devicewould be physically submitted at each step of the process to theExaminer or Institution as may be appropriate. Although not as seamlessa solution, this would offer a secure mechanism for protecting thecontent when only exposing it to the network when it is connected. Awired or wireless method could be used to access and update the contentstored on the device.

Embodiments of the invention are designed to provide deterrent,preventative, detective and corrective controls, ensuring that crucialdata is kept private, confidential and secure. Embodiments of theinvention provides the educational and training programs greaterefficiency, effectiveness and empirical evidence that a student hasachieved an acceptable degree of performance. The student's portfolionecessarily also contains a great deal of private and personalinformation. This invention must guard the content against loss,corruption and access by unauthorized or unknown entities. The solutionprovides mechanisms that deliver a verifiable degree of assurance thatthis is the case. Credentials used by authorized users leverage theentities own Active Directory and LDAP services to ensure thatcredentials have been previously and securely vetted. The registrationprocess for devices and participants applies system level codes whichuniquely identify every participant and their devices. The design thatincludes distributed storage in multiple locations and a file storagestructure that is also distributed makes accessing the content much moredifficult by external threats without the algorithms used internally tore-constitute that content. The content is authenticated at every stageand protected whether at rest or in transit. Anytime a piece of data isaccessed detailed records are kept in the audit trail of when, what, whoand how content was accessed. The level of integration of this inventionand its workflow design are key factors in being able to provide asecure and unified solution which protects content with degree ofveracity acceptable to the justice system.

The workflow, live capture of evidence and proof that a student hasachieved the desired level of proficiency offers improvements in theefficiency and effectiveness of the training and education ofprofessionals. To that end the inclusion of techniques and mechanismsfor measuring the performance of the student is critical to providingexaminers and supervisors with replay and review of the actual procedureor protocol demonstrated by the student. Further the empirical evidenceof that progress can be measured in numerous customizable ways asprovided by a dashboard component of this solution. This dashboard,whether on a smartphone, tablet, or laptop is specific to theparticipant's interests. The dashboards present statistical results ofdata analytics in a way that is conducive to fast and easyinterpretation. The real-time nature of these statistics provides aquick way for both student and examiner to understand what progress isbeing made, to what extent and in comparison on multiple levels.Comparisons could be on a class level, faculty level, institutionallevel or national level. Knowing exactly where one stands in comparisonto classmates makes identifying remedial actions faster and better. Thedata analytics algorithms can be custom designed to match the criteriaof any profession and the performance criteria acceptable to thatinstitution. The content captured over the course of a student'seducation and training offers numerous opportunities to analyze resultsagainst a plethora of data banks. Data analytics of basic informationsuch as grades, averages, means, outliers speed, accuracy, can bemeasured in great detail. Comparisons generated through the dataanalytics engine can produce relevant historical trends and project tofuture performance levels. Although in its infancy the extension ofthese techniques and algorithms to video analytics offers a naturalextension to measuring performance which does not exist today withcurrent professional education and training solutions. The use of videoanalytics to determine the relative emotional and physical state of astudent could be used to determine factors influencing their success.These video analytic techniques can be used to determine whether theexecution of a procedure or protocol can be comfortably achieved by thestudent. This would be particularly true in the training of surgeons orpilots for example. The content captured during the students trainingand education provides a new level of analysis not offered intraditional and current techniques. The lab notes, training notes orpost-operative notes are greatly enhanced and provide much richercontent for those learning when compared with traditional techniques.All of these empirical data can are displayed in visually pleasingdashboard displays perfectly relevant to the participant. Today'spaper-based or online programs do not and cannot offer the same degreeof rich and advanced success factors as does this workflow andtechnology combination.

Although the algorithms described above including those with referenceto the foregoing flow charts have been described separately, it shouldbe understood that any two or more of the algorithms disclosed hereincan be combined in any combination. Any of the methods, algorithms,implementations, or procedures described herein can includemachine-readable instructions for execution by: (a) a processor, (b) acontroller, and/or (c) any other suitable processing device. Anyalgorithm, software, or method disclosed herein can be embodied insoftware stored on a non-transitory tangible medium such as, forexample, a flash memory, a CD-ROM, a floppy disk, a hard drive, adigital versatile disk (DVD), or other memory devices, but persons ofordinary skill in the art will readily appreciate that the entirealgorithm and/or parts thereof could alternatively be executed by adevice other than a controller and/or embodied in firmware or dedicatedhardware in a well known manner (e.g., it may be implemented by anapplication specific integrated circuit (ASIC), a programmable logicdevice (PLD), a field programmable logic device (FPLD), discrete logic,etc.). Also, some or all of the machine-readable instructionsrepresented in any flowchart depicted herein can be implemented manuallyas opposed to automatically by a controller, processor, or similarcomputing device or machine. Further, although specific algorithms aredescribed with reference to flowcharts depicted herein, persons ofordinary skill in the art will readily appreciate that many othermethods of implementing the example machine readable instructions mayalternatively be used. For example, the order of execution of the blocksmay be changed, and/or some of the blocks described may be changed,eliminated, or combined.

It should be noted that the algorithms illustrated and discussed hereinas having various modules which perform particular functions andinteract with one another. It should be understood that these modulesare merely segregated based on their function for the sake ofdescription and represent computer hardware and/or executable softwarecode which is stored on a computer-readable medium for execution onappropriate computing hardware. The various functions of the differentmodules and units can be combined or segregated as hardware and/orsoftware stored on a non-transitory computer-readable medium as above asmodules in any manner, and can be used separately or in combination.

While particular implementations and applications of the presentdisclosure have been illustrated and described, it is to be understoodthat the present disclosure is not limited to the precise constructionand compositions disclosed herein and that various modifications,changes, and variations can be apparent from the foregoing descriptionswithout departing from the spirit and scope of an invention as definedin the appended claims.

What is claimed is:
 1. A system for collecting and evaluating datarecords, the system comprising: a configuration device associated with afirst user, the configuration device being registered with a server andtransmitting a task to the server over a network; a data collectiondevice associated with a second user, the data collection device beingregistered with the server and receiving the task from the server andrecording a data record in response to the task, the data collectingdevice grouping the data record with metadata to produce a verified datarecord and transmitting the verified data record to the server; and adata monitoring device associated with a third user, the data monitoringdevice being registered with the server and receiving the verified datarecord from the server and facilitating the evaluation of the verifieddata record, the evaluation of the verified data record being used togenerate an updated verified data record comprising evaluation resultsand updated metadata, the data monitoring device transmitting theupdated verified data record to the server, the server storing theupdated verified data record in a portfolio.
 2. The system of claim 1wherein the identity of the configuration device and the first user isverified before allowing the transmission of the task to the server; theidentity of the data collection device and the second user is verifiedbefore allowing the reception of the task from the server and beforeallowing the transmission of the verified data record to the server; andthe identity of the data monitoring device and the third user isverified before allowing the reception of the verified data record fromthe server and before allowing the transmission of the updated verifieddata record to the server.
 3. The system of claim 1 wherein the metadatacomprises a plurality of parameters, the parameters being used toconfirm that the verified data record is an accurate representation ofthe data record.
 4. The system of claim 1 wherein data is divided into aplurality of segments to be transferred between the server and, theconfiguration device, the data collection device, or the data monitoringdevice, the receiving device confirming the veracity of each of theplurality of segments as it is received.
 5. The system of claim 4wherein a first algorithm is used to determine the veracity of each ofthe plurality of segments and a second algorithm is use to determine theveracity of all of the plurality of segments after all of the pluralityof segments have been received.
 6. The system of claim 1 wherein themetadata comprises a plurality of parameters, the parameters being usedto confirm that the verified data record is an accurate representationof the data record.
 7. The system of claim 1 wherein the data recordcomprises video recordings, the video recording being used by the thirduser to evaluate the emotional or the physical state of the second user.8. The system of claim 1 wherein the server comprises a plurality ofcontent storage locations, the plurality of content storage locationsimplementing a distributed file structure wherein the portfolio isstored in more than one of the plurality of content storage locations.9. A method for collecting and evaluating academic performance, themethod comprising: registering a student and a personal device with aserver, the student being associated with the personal device;registering an instructor and an instructor device with the server, theinstructor being associated with the instructor device; registering aninstructor and a monitoring device with the server, the instructor beingassociated with the monitoring device; the instructor device combining atask with task metadata to produce a verified task and transmitting theverified task to the server; the personal device receiving the verifiedtask from the server, the personal device recording the performance ofthe verified task to produce a data record, the personal devicecombining the data record with data metadata to produce a verified datarecord and transmitting the verified data record to the server; and themonitoring device receiving the verified data record from the server,the monitoring device being used to record an evaluated data record andcombining the verified data record and the evaluated date record withevaluation metadata to produce a verified evaluated data record andtransmitting the verified evaluated data record to the server, theverified evaluated data record being combined with student results andadded to a portfolio associated with the student.
 10. The method ofclaim 9 further comprising: registering an authority and an authoritydevice with the server, the authority being associated with theauthority device; and the authority device receiving the portfolio fromthe server, the authority device being used to record an overallevaluation of the student's performance, the overall performance beingcombined with performance metadata and with the portfolio to produce averified portfolio and transmitting the verified portfolio to theserver.
 11. The method of claim 9 further comprising: registering anauthority and an authority device with the server, the authority beingassociated with the authority device; and the authority device receivingthe portfolio from the server, the authority device being used to recordan overall evaluation of the instructor's performance, the overallperformance being combined with performance metadata to produce averified instructor evaluation and transmitting the verified instructorevaluation to the server.
 12. The method of claim 9 further comprising:registering an authority and an authority device with the server, theauthority being associated with the authority device; and the authoritydevice receiving the portfolio from the server, the authority devicebeing used to record an overall evaluation of the instructor'sperformance, the overall performance being combined performance metadatato produce a verified instructor evaluation
 13. The method of claim 9wherein the identity of the personal device and the student is verifiedbefore allowing the transmission of the verified data record to theserver; the identity of the monitoring device and the instructor isverified before allowing the reception of the verified data record fromthe server and before allowing the transmission of the verifiedevaluated data record to the server; and the identity of the datainstructor device and the instructor is verified before allowing thebefore allowing the transmission of the verified task to the server. 14.The method of claim 9 wherein data is divided into a plurality ofsegments to be transferred between the server and, the personal device,the instructor device, or the monitoring device, the receiving deviceconfirming the veracity of each of the plurality of segments as it isreceived.
 15. The method of claim 14 wherein a first algorithm is usedto determine the veracity of each of the plurality of segments and asecond algorithm is use to determine the veracity of all of theplurality of segments after all of the plurality of segments have beenreceived.
 16. The method of claim 9 wherein the task metadata, theevaluation metadata, and the data metadata each comprise a plurality ofparameters, the parameters being used to confirm that the verified datarecord is an accurate representation of the data record.
 17. The methodof claim 9 wherein the data record comprises video recordings, the videorecording being used to evaluate the emotional or the physical state ofthe student.
 18. The method of claim 9 wherein the server comprises aplurality of content storage locations, the plurality of content storagelocations implementing a distributed file structure wherein theportfolio is stored in more than one of the plurality of content storagelocations.